Lucene in staged environments

Author: Alexander Tsvirchkov
Posted: 1/21/2008 7:01:17 PM

Question : How can I add a Lucene index for the web database on a runtime (slave) server in Sitecore CMS v5.3.1 (5.3.2)?

Version : Sitecore 5.3.1 rev. 071114 and Sitecore 5.3.2 rev. 071220.

Answer :

Note: Lucene works on the slave server in a staged setup with multiple web farm servers or on NLB if slave servers share the same Web database now. You can use a special web.config parameter:

<!-- INDEX PROPERTIES PER SERVER

Indicates if server specific keys should be used for property values (such as 'last updated').

Default value: false

-->

The logic of updating an index in 5.3.1 rev. 071114 has been reconsidered. Lucene indexes are updated right after adding any entry to HistoryEngine (to the History table of the appropriate database). As soon any entry (any record) is added to HistoryEngine(History table) the IndexingManager.HistoryEngine_AddedEntry event is fired and starts the UpdateIndexAsync job which in turn updates Lucene indexes. The job is added to a queue and executed one-by-one in Sync mode in reality. The job indexes only add new or changed items beginning from the last indexing date. Please see the Properties table and the record with the key value IndexingProvider_LastUpdate.

For the safe side there is another way when Sitecore updates Lucene indexes. Please see the web.config parameter:

So, all Lucene indexes for Master, Core and other databases are rebuilt every 5 minutes according to the appropriate database HistoryEngine ( <CURENT_DATABASE>->History ).

If you have a look at the Master’s database web.config parameter you will see the following entries:

:..

<Engines.HistoryEngine.Storage>

</param>

</obj>

</Engines.HistoryEngine.Storage>

<Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>

The class SqlServerHistoryStorage adds the history of changing, adding and deleting items to the <CURENT_DATABASE>->History table. Please look at the Master database->History table as for example.

The items’ history is used by a scheduled task which updates Lucene indexes in a separate thread every 5 minutes:

<!-- INDEX UPDATE INTERVAL

Gets the interval between the IndexingManager checking its queue for pending actions.

Default value: "00:05:00" (5 minutes)

-->

<setting name="Indexing.UpdateInterval" value="00:05:00" />

All Lucene indexes for Master, Core and other databases will be rebuilt every 05 minutes. As a result it is possible to adjust the updating index for the Web database automatically on the slave server.

Please follow these steps:

Add these entries on the Master server for production(web) database which is shared between Slave and Master servers:
    
      <database id="production" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
        <param desc="name">$(id)</param>
        <securityEnabled>true</securityEnabled>
        <dataProviders hint="list:AddDataProvider">
          <dataProvider ref="dataProviders/main" param1="$(id)">
            <disableGroup>publishing</disableGroup>
            <prefetch hint="raw:AddPrefetch">
              <sc.include file="/App_Config/Prefetch/Common.config" />
              <sc.include file="/App_Config/Prefetch/Web.config" />
            </prefetch>
          </dataProvider>
        </dataProviders>
        <proxiesEnabled>false</proxiesEnabled>
        <proxyDataProvider ref="proxyDataProviders/main" param1="$(id)" />
        
        <Engines.HistoryEngine.Storage>
          <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.$(database)">
            <param desc="connection" ref="connections/$(id)">
            </param>
            <EntryLifeTime>30.00:00:00</EntryLifeTime>
          </obj>
        </Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
        
        <cacheSizes hint="setting">
....
See the added entries in bold above.
Add these entries for the same Production database on the Slave server:
      
      <database id="production" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
        <param desc="name">$(id)</param>
        <securityEnabled>true</securityEnabled>
        <dataProviders hint="list:AddDataProvider">
          <dataProvider ref="dataProviders/main" param1="$(id)">
            <disableGroup>publishing</disableGroup>
            <prefetch hint="raw:AddPrefetch">
              <sc.include file="/App_Config/Prefetch/Common.config" />
              <sc.include file="/App_Config/Prefetch/Web.config" />
            </prefetch>
          </dataProvider>
        </dataProviders>
        <proxiesEnabled>false</proxiesEnabled>
        <proxyDataProvider ref="proxyDataProviders/main" param1="$(id)" />
         
         <indexes hint="list:AddIndex">
           <index path="indexes/index[@id='system']" />
         </indexes>
         <Engines.HistoryEngine.Storage>
           <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.$(database)">
             <param desc="connection" ref="connections/$(id)">
             </param>
             <EntryLifeTime>30.00:00:00</EntryLifeTime>
           </obj>
         </Engines.HistoryEngine.Storage>
         <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
         
Here we added the index for the production database in additional.
Perform a full publish from the Master to the Slave server. It will update the production database (web) that is shared between Master and Slave servers.
On the Slave server create the Lucene index for the production (web) database via Control Panel-> Rebuild Search Indexes. It will create the index for the production database.
The last updating date of the Lucene index is saved in the <Current_Database>-> Properties table with the key = IndexingProvider_LastUpdate. In our case it is saved in the production database.
Note:UTC date is used.

Related reading:

Prev Next